Personalized entity repository

ABSTRACT

Systems and methods are provided for a personalized entity repository. For example, a computing device comprises a personalized entity repository having fixed sets of entities from an entity repository stored at a server, a processor, and memory storing instructions that cause the computing device to identify fixed sets of entities that are relevant to a user based on context associated with the computing device, rank the fixed sets by relevancy, and update the personalized entity repository using selected sets determined based on the rank and on set usage parameters applicable to the user. In another example, a method includes generating fixed sets of entities from an entity repository, including location-based sets and topic-based sets, and providing a subset of the fixed sets to a client, the client requesting the subset based on the client&#39;s location and on items identified in content generated for display on the client.

BACKGROUND

The use of mobile devices, such as smart phones, wearable devices,tablets, laptops, etc., has been increasing. By understanding thecontent viewed by and actions taken by a user, the mobile device canbuild a much better user experience, for example by offeringpersonalized predictions and assistance to the user. Part ofunderstanding the content and actions involves identifying andclassifying entities recognized in the content of the device screen. Theentities may exist in an entity repository, such as a knowledge base orvision model, which stores facts and information about entities. Large,public entity repositories may include millions of such entities. Mostcomputing devices, especially mobile computing devices, such as smartphones and tablets, have limited storage and use the entity repositoryvia a connection with a server.

SUMMARY

Implementations divide an entity repository into several fixed sets orslices. A set may be location-based, topic-based, action-based, orfunctional. The system may determine the sets at a server and provideonly those sets to a client device that are relevant to that particularclient. The sets downloaded to the client are a personalized entityrepository that can be accessed without connection to the server. Theclient device may include a set identification engine. The setidentification engine may include a prediction model that predicts oneor more sets given some text, images, or other features. The predictionmodel may be used to determine which sets are most beneficial to theuser. The set identification engine may also track a location of theuser to determine which location sets may be most relevant.

Sets may be ranked for the user of the client device, e.g., based on theuser's location, a user's search history, content the user has beenviewing on the device, time of day, signals from other devices, etc.Thus, for example, if a user begins researching a trip to Hawaii, thesystem may determine that a set with entities related to Hawaii isparticularly relevant to the user. If a user flies from New York to LosAngeles, the system may determine location sets for Los Angeles arerelevant. The client device may use the rankings to determine which setsto retrieve and which to delete from memory. For example, if a setpreviously downloaded is no longer relevant, the system may remove thatset to make room for another set. The system may employ set usageparameters to determine which sets to include in the personalized entityrepository at any time. In some implementations, the user may establishand control the set usage parameters. Updating a set may includedetermining a delta for the set to reduce the data transferred to theclient device. In some implementations, the sets may be versioned, e.g.,so that a schema change does not break the applications using the model.

According to certain aspects of the disclosure, a mobile devicecomprises a display device, a personalized entity repository stored inmemory, the personalized entity repository including a plurality offixed sets of entities from an entity repository stored at a server,wherein each fixed set has a respective identifier and includesinformation about the entities in the set, at least one processor, andmemory storing instructions that, when executed by the at least oneprocessor, cause the mobile device to perform operations. The operationinclude identifying fixed sets of the entity repository that arerelevant to a user of the mobile device based on context associated withthe mobile device, ranking the fixed sets by relevancy, determiningselected sets from the identified fixed sets using the rank and setusage parameters applicable to the user, and updating the personalizedentity repository using the selected sets.

According to certain aspects of the disclosure, a method includesreceive a screen captured image configured to display content on adisplay of a mobile device, determining text in the image by performingtext recognition on the image, providing the text to a set predictionmodel, wherein the set prediction model is trained to predict one ormore fixed sets of entities, and storing at least one fixed set ofentities of the predicted fixed sets of entities in a personalizedentity repository in memory on the mobile device.

According to certain aspects of the disclosure, a method may includegenerating a plurality of fixed sets of entities from an entityrepository. The fixed sets include location-based sets, eachlocation-based set including entities from the entity repository thathave a location inside a cell, the cell being associated with the set,and topic-based sets, at least some of the topic-based sets includingentities from the entity repository that are related to each other viaan embedding similarity. The method may also include providing a subsetof the fixed sets of entities to a client device, the client devicerequesting the subset based on a location of the client device and onrecognized items identified in content generated for display on theclient device.

In one general aspect, a computer program product embodied on acomputer-readable storage device includes instructions that, whenexecuted by at least one processor formed in a substrate, cause acomputing device to perform any of the disclosed methods, operations, orprocesses. Another general aspect includes a system and/or a method forgenerating fixed sets of entities in an entity repository and providingsome of the fixed sets to a personal computing device as a personalentity repository, substantially as shown in and/or described inconnection with at least one of the figures, and as set forth morecompletely in the claims.

One or more of the implementations of the subject matter describedherein can be implemented so as to realize one or more of the followingadvantages. As one example, generating fixed sets of entities enablesthe system to provide personal or custom entity repositories in ascalable way. The customized entity repository can aid in on-device textanalysis and image analysis to support user assistance capabilities evenwithout connectivity to a network. The user may control the amount ofresources dedicated to the personal entity repository and the system mayautomatically determine which sets best utilize the allocated resources.As another example, the system may predict which sets are most relevantto a user. The prediction may be based on a number of factors specificto the user and/or the personal computing device, such as location, timeof day, signals from other computing devices associated with the user orknown to the device, recent activity on the computing device, etc. Asanother example, implementations may support versioning of the sets, sothat a change that restructures the information in the set does notbreak applications that use the set.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features will beapparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example system in accordancewith the disclosed subject matter.

FIG. 2 illustrates an example display of a user interface for specifyingset usage parameters for a personalized entity repository, in accordancewith the disclosed subject matter.

FIG. 3 illustrates an example display of content that provides contextfor determining which sets are most relevant to a user of the computingdevice, in accordance with disclosed implementations.

FIG. 4 illustrates an example display of a user interface for suggestingadditional sets for inclusion in a personalized entity repository, inaccordance with disclosed implementations.

FIG. 5 illustrates another example display of a user interface forsuggesting additional sets for inclusion in a personalized entityrepository, in accordance with disclosed implementations.

FIG. 6 illustrates a flow diagram of an example process for generatingand updating fixed sets of entities, in accordance with disclosedimplementations.

FIG. 7 illustrates a flow diagram of an example process for building apersonal entity repository using fixed sets of entities, in accordancewith disclosed implementations.

FIG. 8 illustrates a flow diagram of an example process for identifyingfixed sets of entities relevant to a user of a client device, inaccordance with disclosed implementations.

FIG. 9 illustrates a flow diagram of an example process for selectingfixed sets of entities from fixed sets of entities relevant to a user ofa client device based on set usage parameters, in accordance withdisclosed implementations.

FIG. 10 shows an example of a computer device that can be used toimplement the described techniques.

FIG. 11 shows an example of a distributed computer device that can beused to implement the described techniques.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

FIG. 1 is a block diagram of a personal entity repository system inaccordance with an example implementation. The system 100 may be used tobuild a personalized entity repository from selections of pre-computedor fixed sets of entities. The sets are fixed in the sense that they arecomputed ahead of any particular request for a set, or in other wordsare not generated in response to a request for a set. The system 100 maydetermine which sets are most useful at a particular point in time andmay update the personalized entity repository to ensure entities in thepersonalized repository are relevant to the user. Updating thepersonalized entity repository includes deleting sets that are lessrelevant to the user now than before and adding new sets that havebecome relevant. The personalized entity repository may be used todetermine entities in content generated for display on a computingdevice to personalize a user experience on the computing device.Personalizing the user experience can include predicting actions,topics, words, or phrases, etc. The system 100 in FIG. 1 is illustratedas a client-server system, with some data processing or data storageoccurring at a server 110 and other data processing occurring at theclient device 150. However, other configurations and applications may beused and implementations are not limited to the exact configurationsillustrated.

The personalized entity repository system 100 may include a server 110,which may be a computing device or devices that take the form of anumber of different devices, for example a standard server, a group ofsuch servers, or a rack server system. For example, server 110 may beimplemented in a distributed manner across multiple computing devices.In addition, server 110 may be implemented in a personal computer, forexample a laptop computer. The server 110 may be an example of computerdevice 900, as depicted in FIG. 9 , or computer device 1100, as depictedin FIG. 11 . Server 110 may include one or more processors formed in asubstrate configured to execute one or more machine executableinstructions or pieces of software, firmware, or a combination thereof.The processors can be semiconductor-based that is, the processors caninclude semiconductor material that can perform digital logic.

The server 110 may store or have access to entity repository 130. Theentity repository 130 may store information about entities. An entitymay be may be a person, place, item, idea, topic, word, phrase, abstractconcept, concrete element, other suitable thing, or any combination ofthese. In some implementations, the entity repository 130 may be aknowledge base, which stores information about entities. In someimplementations, a knowledge base stores information about entities inthe form of relationships between entities. For example, entities in aknowledge base may be related to each other by labeled edges thatrepresent relationships. The knowledge base may also store attributes ofan entity. Some knowledge bases are large, sometimes including millionsof entities. A knowledge base with a large number of entities and even alimited number of relationships may have billions of connections.

The entity repository 130 may also include image recognition entitiesthat can be used to identify an entity in an image. For example, theentity repository 130 may include entities that represent known imagesof the entity and/or characteristics of the image and are used in imagerecognition processes (e.g., using image similarity techniques) toidentify entities in images. For example, the characteristics may befeatures provided to an image recognition model (e.g., machine learningalgorithm). The entity repository 130 may represent a single knowledgebase, a combination of distinct knowledge bases, image recognitionentities, and/or a combination of a knowledge base and imageinformation. In some implementations, entity repository 130 may bestored in an external storage device accessible to server 110. In someimplementations, the entity repository 130 may be distributed acrossmultiple storage devices and/or multiple computing devices, for examplemultiple servers. The entities and relationships in a knowledge base maybe searchable, e.g., via an index. For example, the index may includetext by which an entity has been referred to. Thus, reference to theknowledge base may be understood to include an index that facilitatesfinding an entity using a text equivalent.

The server 110 can also include one or more computer memories. Thememories, for example, a main memory, may be configured to store one ormore pieces of data, either temporarily, permanently, semi-permanently,or a combination thereof. The memories may include any type of storagedevice that stores information in a format that can be read and/orexecuted by the one or more processors. The memories may includevolatile memory, non-volatile memory, or a combination thereof, andstore modules or engines that, when executed by the one or moreprocessors, perform certain operations. In some implementations, themodules may be stored in an external storage device and loaded into thememory of server 110. In some implementations, the modules may includeentity set generation engine 120. The entity set generation engine 120may divide entities in an entity repository 130 into fixed entity sets132. Each set in the fixed entity sets 132 may be thought of as a sliceof the entity repository 130 that includes entities related by somecriteria. In some implementations a slice may contain metadata about anentity (e.g., name, description, canonical image), relationships withother entities, and/or information used to detect the entity in text,such as common names, aliases, abbreviations, nicknames, and othersignals. Each fixed set of entities in the fixed entity sets 132 mayhave an identifier so that a particular set can be tracked, identified,and requested by applications or client devices.

In some implementations, the entity set generation engine 120 maygenerate various types of sets, such as location-based sets, topic-basedsets, or functional sets. A location-based set may include entitiesassociated with a particular geographic cell. A geographic cell is anarea of the surface of the earth defined by boundaries, for example twolines of latitude and two lines of longitude. Such cells are common formap applications, such as GOOGLE MAPS. A location-based fixed set ofentities may include any entities known to be located within the cell.The entity repository 130 may include an attribute that specifies alocation for an entity or may include metadata identifying the location.For example, a restaurant entity in entity repository 130 may have alocation attribute specifying a particular geo cell or cells orlatitude/longitude coordinates. While geo cells, i.e., cells used in amap application, are typically of similar size, location-based entitysets need not correspond to one geo cell but may be based, instead, onan area having a specified number of entities located within the area.Thus, for instance, a cell for a location-based set of entities in NewYork City may be quite small (e.g., ten square miles) because it isdensely populated while a cell for a location-based set of entities inwestern Wyoming may be hundreds of square miles but have the same numberof entities as the cell in New York City. Thus, for example, the entityset generation engine 120 may merge adjacent geo cells until thequantity of entities associated with the merged cell reaches a minimumquantity and fauns a location-based cell for a fixed set of entities.

The entity set generation engine 120 may also generate topic-based fixedsets of entities. In some implementations, the entity set generationengine 120 may generate topic-based sets by clustering the entitiesusing conventional clustering techniques based on how related entitiesare. For example, the clustering may be based on characteristics of theentities, e.g., embeddings, trained on search records or on text such astext from documents indexed for a search engine. In someimplementations, the entity set generation engine 120 may be given aseed entity and may form a cluster based on the seed entity. In someimplementations, one or more topic-based sets may be based oncollections identified in the entity repository. For instance, entitiesin the entity repository 130 may include an indication of one or morecollections that the entity belongs to or the entity repository 130 mayinclude a collection definition, or in other words requirements forinclusion in the collection, and entities may be evaluated against therequirements to determine membership in the collection. The entity setgeneration engine 120 may use entity membership in the collection togenerate one or more of the topic-based sets. In some implementations,the entity set generation engine 120 may compute an embedding for eachtopic-related slice. The embedding may be used to compute similaritywith a query embedding, e.g., generated by a set identification engine,to determine which sets are most similar to the query embedding. Forexample, the embedding may represent a cluster center.

The entity set generation engine 120 may also generate functional setsof entities. A functional set may include entities that are deemed mostpopular, e.g., entities that are most frequently searched for oraccessed, for example based on appearance in search queries. Afunctional set may also be based on a capability, for example thecapability to translate from one language to another. As anotherexample, a functional set may include entities likely to be encounteredwhen using or requested by a particular application. Of course, thesefunctional sets are provided as examples only and functional entity setsmay include any collection of entities common to a particular purpose,characteristic, or function. In some implementations, the entity setgeneration engine 120 may generate fixed sets of entities that arecombinations of the above. For example, the entity set generation engine120 may generate a fixed set of entities for restaurants in San JoseCalifornia or another set for artwork recognition in Paris France. Insome implementations, the entity set generation engine 120 may include auser interface that enables a user to curate one or more fixed sets ofentities.

The entity set generation engine 120 may also update the fixed sets ofentities 132 periodically. For example, as entities are added, deleted,or updated in the entity repository 130 the entity set generation engine120 may determine changes to the fixed sets. In some implementations,the entity set generation engine 120 may determine deltas for each set,for example a list of entities to be deleted from the fixed set and alist of entities to be added to the fixed set. The entity set generationengine 120 may provide the deltas to the client device 150, either as apush or a pull. The deltas reduce the amount of data transmitted betweenthe server 110 and the client device 150 while still enabling the clientdevice 150 to have the most current entity sets. This is especiallyuseful when a particular fixed entity set changes often, such as a setof entities that represents movies. In some implementations, the entityset generation engine 120 may also version each of the fixed sets, alsoreferred to as slices. For example, the entity set generation engine 120may change the data format of the slices or may change the slices inanother way that breaks the schema. When the schema is broken theapplications that uses the slices may need a corresponding update to usethe new slices. Accordingly the entity set generation engine 120 mayversion the fixed sets so that applications that have not been updatedmay continue to use older versions without malfunctioning. In someinstances, the client device 150 may include two versions of the samefixed slice (e.g., a slice representing a particular location), untilapplications that use the old schema are updated.

In some implementations, the entity set generation engine 120 may alsotrain and maintain a set prediction model 122. A set prediction model,such as set prediction model 122, may be any type of machine-learningalgorithm, such as a long short-term memory (LSTM) neural network,feed-forward neural network, a support vector machine (SVM) classifier,etc., that can predict a fixed set of entities given a set of featuresor signals. The features may include a device location, text or imagesin content (e.g., documents, search records, etc.), applicationstypically used for various tasks, etc. As a machine-learning algorithm,the set prediction model has two modes, a training mode and an inferencemode. For example, in the training mode the set prediction model 122 mayuse labeled search records 134 and labeled crawled documents 136 topredict one or more fixed sets of entities from fixed entity sets 132given a set of signals. The labels may be created automatically or byhuman experts. The signals may be taken from content generated on aclient device, such as device 150 or from search records, such as searchrecords 134 or user-specific records such as screen capture index 172.In a training mode the set prediction model 122 may also take in alocation as a signal (e.g., current geo cell location, coordinates,etc.) to predict a set of entities for the given location. Duringtraining the parameters of the model are updated to better reflect thelabels assigned to text in the training documents. In an inference mode,the set prediction model 122 will predict one or more fixed entity setsfrom fixed entity sets 132 in response to a set of features thatrepresents the signals. Each predicted set may have an associatedconfidence score or probability score indicating a level of certaintythat the features provided predict the particular fixed set of entities.In some implementations, the confidence score may be based on asimilarity measure which may differ depending on the type of set. Forexample, the confidence score for a location-based sets may be based onphysical distance from a specified location, e.g., the current locationof a computing device. The confidence score for a topic-based set may bebased on an embedding distance with a query, such as an embeddinggenerated based on signals from a client device. Such signals caninclude text recently seen on the screen, the state or proximity ofexternal devices, content of recent searches, stated user interests, anapplication installed or executing on the client device, a time stamp,etc.,

Once trained, and then on a periodic basis to account for updates, theentity set generation engine 120 may provide the set prediction model122 to the client device 150. The client device 150 may store the modelas set prediction model 164. In some implementations, the client device150 may personalize the set prediction model 164 by performing furthertraining. The training may use information from search recordsassociated with the user, for example from search records stored atclient device 150 or screen capture index 172. Thus the set predictionmodel 164 may be a copy of set prediction model 122 or a personalizedcopy of set prediction model 122.

The server 110 may include search records 134 and crawled documents 136.The search records 134 may include search logs, aggregated data gatheredfrom queries, or any other data based on queries. In someimplementations, the search records 134 may be generated by a searchengine in the normal process of generating search results. In someimplementations, the search records 134 may be stored on a differentcomputing device that is accessible to server 110. In someimplementations, the search records may be distributed across aplurality of computing devices. The crawled documents 136 may bedocuments obtained using known or later developed web-crawlingtechniques, for example. In some implementations, the crawled documents136 represent documents available over the Internet and may be anindexed form of the documents.

The personalized entity repository system 100 may include a computingdevice 150. Computing device 150 may be any mobile computing device,such as a smartphone or other handheld computing device, a tablet, awearable computing device, etc., that operates in a closed mobileenvironment rather than a conventional open web-based environment.Computing device 150 may also be other types of personal electroniccomputing devices, such as a laptop or net-based computer, a desktopcomputer, a television with a processor, etc. Computing device 150 maybe an example of computer device 1000 or 1050, as depicted in FIG. 10 .Computing device 150 may be a computing device used by a single user, orcan be a computing device shared by multiple users.

Computing device 150 may include one or more processors formed in asubstrate configured to execute one or more machine executableinstructions or pieces of software, firmware, or a combination thereof.The processors can be semiconductor-based—that is, the processors caninclude semiconductor material that can perform digital logic. Thecomputing device 150 may thus include one or more computer memoriesconfigured to store one or more pieces of data, either temporarily,permanently, semi-permanently, or a combination thereof. The computingdevice 150 may thus include applications 155, which represent machineexecutable instructions in the form of software, firmware, or acombination thereof. The components identified in the applications 155may be part of the operating system or may be applications developed torun using the operating system. In some implementations, applications155 may be mobile applications. Conventionally, mobile applicationsoperate in a closed environment, meaning that the user employs separateapplications to perform activities conventionally performed in aweb-based browser environment. For example, rather than going tobookit.com to book a hotel, a user of the computing device 150 can use amobile application in applications 155 provided by bookit.com.Applications 155 may also include web applications, which may mirror themobile application, e.g., providing the same or similar content as themobile application. In some implementations, the applications 155 mayinclude functions performed by an operating system of the computingdevice 150.

The applications 155 may include a screen content agent 160 and a setidentification engine 162. In some implementations, one or more of theseapplications can be provided by the operating system (not shown) of thecomputing device 150. In some implementations, one or more of theseapplications can be downloaded and installed by the user.

The screen content agent 160 can include various functionalities. Insome implementations, the screen content agent 160 may be configured toget textual information represented on the screen of the computingdevice from an application program interface (API). In someimplementations, the screen content agent 160 may be built into theoperating system, which can determine the content of text fieldsdisplayed on the screen. The textual information may be consideredscreen captured content, and each call to the API or each time thecontent of text fields is determined may be considered a screen capture.In some implementations, the screen content agent 160 may be configuredto capture the image displayed on the screen by copying or reading thecontents of the device's frame buffer. Thus, the captured screen may bean image and may be referred to as a captured image. The screen contentagent 160 may capture the screen at intervals. The interval can besmall, for example every half second or every second. In someimplementations, the screen content agent 160 may be configured tocapture the screen every time a touch event occurs (e.g., every time theuser touches the screen to scroll, zoom, click a link etc.), in responseto an explicit user request or command, or when the device transitionsfrom one mobile application to another mobile application. In someimplementations, the screen content agent 160 may increase the intervalat which a screen capture occurs when the screen does not change. Inother words, when the screen is static, the screen content agent 160 maycapture images less often.

The screen content agent 160 may provide the captured content or screenimages and metadata to a recognition engine, which may be part of thescreen content agent 160 and located on the computing device 150. Insome implementations, the recognition engine may be located at a server,such as server 110. When a screen capture image is provided to therecognition engine, the recognition engine may perform image and textrecognition on the image to identify words, entities, logos, etc. in thecontent of the screen capture image. The recognition engine may beconfigured to perform various types of recognition, such as characterrecognition, image recognition, logo recognition, etc., usingconventional or later developed techniques. Thus, the recognition enginemay generate recognized content, which can be from words as well as fromimages.

The screen content agent 160 may also determine and use metadata aboutthe screen capture image. The metadata may include the timestamp, themobile device type, a mobile device identifier, the mobile applicationrunning when the content was captured, e.g., the application thatrendered the content displayed on the screen, etc. In someimplementations, the metadata may also include which applications areactive, the location of the device, ambient light, motion of the device,etc. In some implementations, the metadata may include signals fromother computing devices. For example, the screen content agent 160 maybe provided or may obtain information from external devices such asappliances, televisions, personal assistants, music devices, alarmsystems, etc., that are configured to communicate with the client device150. For instance, a voice-activated electronic list making device maystore a list of grocery items to be purchased. This list may betransmitted to the client device 150 either directly from the electroniclist making device or via user account information. As another example,a user may have a tablet and a smartphone that share information or ahusband and a wife may have smartphones that share information. Thus, insome implementations, the information available to the client device 150may include information that is provided by other devices. Some or allof this information can be included in metadata associated with a screencapture image.

The system may use the metadata and information obtained via the screencapture image to assist in on-device intelligence that analyzes theinformation to assist the user with tasks performed on the mobiledevice. For example, a user having a conversation with a friend mayinclude the suggestion to see a movie. On-device intelligence mayidentify the suggestion and offer an action for viewing a movie reviewor for buying tickets. The screen content agent 160 may use an entityrepository to determine whether recognized content includes knownentities. While the screen content agent 160 may use a public entityrepository, such as entity repository 130, this requires a connectionwith server 110 and can slow down the recognition and action suggestionprocess. Accordingly, the client device 150 may have a personalizedentity repository 176, stored on the client device 150. The personalizedentity repository 176 may be a collection of fixed sets of entities,which are obtained from the server 110, for example from fixed entitysets 132. The personalized entity repository 176 may be generated andmaintained by the set identification engine 162.

In some implementations, the screen content agent 160 can include anindexing engine configured to index the captured content. The index mayalso associate a screen capture image with the text, entities, images,logos, etc. identified in the image. Thus, for example, the indexingengine may generate index entries (e.g., stored in screen capture index172) for a captured image and captured content. In some implementationsthe indexing engine may be on a server, such as server 110, and thescreen content agent 160 may provide the captured image and capturedcontent to the server. The index may be an inverted index, where a keyvalue (e.g., word, phrase, entity, image, logo, etc.) is associated witha list of images (e.g., copies of the captured screen images) thatinclude the key value. The index may include metadata (e.g., where onthe captured image the key value occurs, a rank for the key value forthe image, etc.) associated with each captured image in the list. Insome implementations, the index may also include a list of capturedimages indexed by a timestamp. The indexing engine may store the indexin memory, for example in screen capture index 172. Of course, in someimplementations the system may store the index in a user account on aserver in addition to or instead of on the computing device 150. Theuser of the computing device 150 may control when the screen contentagent 160 is active. For example, the user may specify that the screencontent agent 160 is active only when other specified applications 155are running (e.g., only when in a social media mobile application). Theuser may also manually turn the screen content agent 160 on and off, forexample via a settings application. In some implementations, the usermay invoke the screen content agent 160 with a gesture or action.Disabling the screen content agent 160 may also disable the predictionand maintenance of the personalized entity repository described herein.

The computing device 150 may also include a set identification engine162. The set identification engine 162 may be configured to determinewhich sets of entities in the fixed entity sets 132 should be includedin the personalized entity repository 176. The set identification engine162 may use information collected or generated by the screen contentagent 160 as well as a set prediction model 164 and set usage parametersto determine which sets are potentially relevant to the user and obtainthose sets from the server 110. For example, the set identificationengine 162 may collect signals as input for the set prediction model164. The signals can include information and metadata such as devicelocation, a time of day, the state of various external devices incommunication with the client device 150, proximity to other devices,information from content in a screen capture image or a series of screencapture images, information from the screen capture index 172 or searchrecords for the user, information in a user profile, etc. As explainedabove, external devices such as appliances, televisions, personalassistants, music devices, alarm systems, etc., may be configured tocommunicate with and provide status information to the client device 150and the status information may be included in the signals generated bythe identification engine 162. The information used to generate signalsmay be stored as metadata in the screen capture index 172 orgenerated/collected at the time the set identification engine 162determines the fixed sets of entities appropriate for the personalizedentity repository 176.

As discussed earlier, the set prediction model 164 may be a copy of setprediction model 122 or may be a personalized version of set predictionmodel 122. When prediction model 164 is a personalized version, the setidentification engine 162 may provide training examples from data storedon the client device 150 or from a user profile associated with the userof client device 150. For example, the training examples may begenerated using the screen capture index 172 or search records for theuser 180. In some implementations, the client device, with userpermission, may provide updates to the set prediction model 122 on theserver 110 so that the set prediction model can learn from predictionsmade over many client devices.

The set identification engine 162 may provide signals, for example as aset of features, to the set prediction model 164 in an inference mode.In response, the set prediction model 164 may then provide theidentifiers of one or more predicted fixed sets of entities to the setidentification engine 162. The set identification engine 162 may thenrank the predicted fixed sets of entities. The client device 150 may bea device with a small form factor, which limits storage space on thedevice. Accordingly, the set identification engine 162 may rank thepredicted fixed sets of entities to determine which fixed sets toinclude in personalized entity repository 176. In some implementations,the set identification engine 162 may work within set usage parametersselected by the user of the client device. The set usage parameters mayinclude a maximum quantity of fixed sets to store on the client device,a maximum amount of storage used by the personalized entity repository176, a minimum rank, or a combination of these. If the quantity ofpredicted sets cannot be accommodated on the client device 150, the setidentification engine 162 may select as many of the highest ranked setsas the set usage parameters accommodate.

In some implementations the ranking may be determined by the setprediction model 164. In some implementations, the set identificationengine 162 may adjust rankings provided by the set prediction model 164.For example, if a particular fixed set is needed by a web applicationthe user accesses often or has just installed, the set identificationengine 162 may boost the ranking of that particular fixed set. The setidentification engine 162 may also use metadata to adjust rank. Forexample, if Alice and Ted are travelling and have provided consent toshare data, the client device for Alice may determine that the clientdevice of Ted already has a particular fixed set included in thepersonalized entity repository on his device. Accordingly, when theAlice's client device detects Ted's device in close proximity, the setidentification engine 162 on Alice's device may demote the ranking ofthat particular fixed set on her device.

In some implementations, the set identification engine 162 may generatethe ranking of predicted fixed sets on a periodic basis or after someevent, such as the install of a new application, the activation of anapplication that has not been used for a predetermined period of time,when the available space changes, e.g., the device becomes low on diskspace (e.g., falls below a specified percentage) or some action thatfrees up a specified percentage of space, when the user changeslocation, etc. The set identification engine 162 may also update thepersonalized entity repository 176 in response to the ranking. Forexample, the set identification engine 162 may update the personalizedentity repository 176 periodically, when an application using aparticular fixed set of entities is installed, or when a rank for afixed set of entities becomes more highly ranked that one of the setscurrently in the personalized entity repository 176. For example, ifAlice begins researching a trip to Hawaii, eventually the set predictionmodel 164 will predict a fixed slice related to Hawaii based on content(e.g., from screen capture index 172 or search records) Alice has beenviewing. Because this set has previously never ranked high enough to beincluded in personalized entity repository 176, once the rank doesexceed the rank of a set currently in the personalized entity repository176 or when a change in rank exceeds some predetermined threshold, theset identification engine 162 may automatically update the personalizedentity repository 176 or may obtain approval from Alice before includingthis particular set. Once Alice returns from Hawaii or if she does notgo and does not access content on Hawaii for a period of time, the rankof the particular slice may fall, e.g., fall below another slice not inthe personalized entity repository 176 or fall a predeterminedpercentage. This may trigger execution of the set identification engine162 so that another slice (i.e., another fixed set of entities) may takeits place in the personalized entity repository 176.

Updating the personalized entity repository 176 may be accomplished in avariety of ways. In some implementations, the entire repository isdeleted and replaced by fixed sets selected based on a rank and the setusage parameters. In some implementations, the set usage parameters maybe set and controlled by the user. In some implementations the setidentification engine 162 may determine whether a particular fixed setthat should be in the personalized entity repository 176 already existsin the personalized entity repository 176. If it does the setidentification engine 162 may either do nothing or determine if the sethas changed, e.g., has been updated at server 110. If an update hasoccurred the set identification engine 162 may download the entire setor may download a delta to apply to the set. The set identificationengine 162 may delete sets from the personalized entity repository 176to make room for sets with a higher rank.

The computing device 150 may be in communication with the server 110 andwith other mobile devices over network 140. Network 140 may be forexample, the Internet, or the network 160 can be a wired or wirelesslocal area network (LAN), wide area network (WAN), etc., implementedusing, for example, gateway devices, bridges, switches, and/or so forth.Network 140 may also represent a cellular communications network. Viathe network 140 the server 110 may communicate with and transmit datato/from computing device 140 and computing device 140 may communicatewith other mobile devices (not shown).

The personalized entity repository system 100 represents one exampleconfiguration and implementations may incorporate other configurations.For example, some implementations may combine one or more of thecomponents of the screen content agent 160, the set identificationengine 162, or the set prediction model 164 into a single module orengine. Similarly, some implementations may combine one or more of theentity set generation engine 120 or the set prediction model 122 into asingle module or application. As another example one or more of the datastores, such as the screen capture index 172, the personalized entityrepository 176, or user profiles on client device 150 or entityrepository 130, fixed entity sets 132, search records 134, or crawleddocument 136, may be combined into a single data store or maydistributed across multiple computing devices, or may be stored atanother location.

To the extent that the personalized entity repository system 100collects and stores user-specific data or may make use of personalinformation, the users may be provided with an opportunity to controlcollection of the user information (e.g., information about a user'ssocial network, social actions or activities, a user's preferences, or auser's current location), or to control whether and/or how to storescreen capture images and content. For example, the system may refrainfrom capturing content for certain applications, such as bankingapplications, health applications, or other similar applications orwhere capturing such content violates terms of service. In addition, theuser may be provided with the opportunity to disable capturing screencontent for specific applications or categories of applications. Inaddition, certain data may be treated in one or more ways before it isstored or used, so that personally identifiable information is removed.For example, a user's identity may be treated so that no personallyidentifiable information can be determined for the user, or a user'sgeographic location may be generalized where location information isobtained (such as to a city, ZIP code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about the user and usedby a personalized entity repository system.

FIG. 2 illustrates an example display of a user interface 200 forspecifying set usage parameters for a personalized entity repository, inaccordance with the disclosed subject matter. The display may be adisplay of a mobile device or other personal computing device, such asclient device 150 of FIG. 1 . In some implementations, the userinterface may be provided by the operating system of the client deviceor may be provided by a particular application, e.g., from applications155 of FIG. 1 . The user interface 200 includes controls 205 for acontent agent, such as content agent 160 of FIG. 1 . The controls 205may regulate whether and how the content agent executes and what actionsthe content agent performs. The user interface 200 may also include setusage parameters for a personalized entity repository. The set usageparameters may include one or more of a maximum storage 210, a maximumnumber of sets 215, or a percent of storage 220. The maximum storage 210may set a limit on the amount of memory used by a personalized entityrepository, such as personalized entity repository 176 of FIG. 1 . Theclient device may download and store fixed sets of entities (slices) butnot to exceed the maximum storage 210.

The set usage parameters may also include maximum sets 215. The maximumsets 215 limits the number of fixed sets of entities that the clientdevice will include in the personalized entity repository. In otherwords, the client device may download and store only a specified numberof sets. The percent of storage 220 may offer a flexible limit on thesize of the personalized entity repository. For example, when the clientdevice has more memory, the percent of storage 220 parameter may allow alarger personalized entity repository, but as space on the client devicediminishes, the parameter may limit the sets stored in the personalizedentity repository. The user interface 200 may also enable the user toselect a combination of set usage parameters, e.g., via checkboxes, toturn on or off set usage parameters. Thus, the user may control the sizeof the personalized entity repository on their device, which in turncontrols which slices are included in the repository. The user interface200 may also include a control 225 that enables the user to specificallyselect one or more sets of entities for inclusion in the personalizedentity model. When the user specifically uses the control 225 to selecta slice of the entity repository, the system may rank the relevancy ofthat slice high, so that it is always included in the personalizedentity model.

FIG. 3 illustrates an example display 300 of content that providescontext for determining which sets are most relevant to a user of thecomputing device, in accordance with disclosed implementations. Thedisplay 300 may be a display of a mobile device or other computingdevice, such as client device 150 of FIG. 1 . In the example of FIG. 3 ,the text “Alien Invaders” is selected. The selection may have been madeby a user or may have been made automatically by the client device(e.g., a content agent or operation suggestion application running onthe client device). The system has provided four suggested operationsfor the selection 305, namely operation 340, which may be for a mobileapplication that provides local movie times, a movie review operation325, and a movie database operation 320. The user interface may haveselected operation 340, 325, and 320 based on identification ofselection 305 as an entity that exists in the personalized entityrepository. Thus, the system may provide display 300 even withoutconnection to a server and to a server-based entity repository.

FIG. 4 illustrates an example display of a user interface 400 forsuggesting additional slices for inclusion in a personalized entityrepository, in accordance with disclosed implementations. The userinterface 400 may be generated on a mobile device or other computingdevice, such as client device 150 of FIG. 1 . In the example of FIG. 4 ,a user has entered text into a search interface 405. Search interface405 may be for a browser or a browser-like mobile application, but couldbe from any search bar in any application. The system may analyze thetext provided to the search interface 405 and determine, e.g., via a setidentification engine using a set prediction model, that the text ishighly relevant to a topic-based fixed set of entities. The highrelevancy may be determined based on a high confidence score orprobability from the set prediction model. The high relevancy may bebased not only on the content of the user interface 400 by also onpreviously presented content. For example, the user may have beenlooking at hotels in Hawaii in a booking application or may have beenreading about things to see in Hawaii. While the client device may nothave an entity for Hawaii in the current personalized entity repository,the set prediction model may recognize several words or images thatpredict the set. In some implementation, the set identification enginemay be continuously analyzing text for relevant models. In someimplementations, the set identification engine may be invoked during useof a search interface.

When the probability of the fixed set of entities indicates highrelevancy, the system may automatically download and store the fixed setof entities for Hawaii, also referred to as the Hawaii slice. In someimplementations, this may be a location-based set for all entitieslocated in Hawaii, or a topic-based set, or a combination of these. Insome implementations, the user interface may include confirmation window410. Confirmation window 410 may enable the user to accept or reject thesuggestion. If the user rejects the suggestion, the system may notdownload the Hawaii slice. If the user accepts the suggestion, thesystem may download the Hawaii slice, and may remove one or more slicescurrently in the personalized entity repository to make room for the newslice, as explained in more detail with regard to FIG. 7 .

FIG. 5 illustrates another example display of a user interface 500 forsuggesting additional sets for inclusion in a personal entityrepository, in accordance with disclosed implementations. The userinterface 500 may be generated on a mobile device or other computingdevice, such as client device 150 of FIG. 1 . In the example of FIG. 5 ,the user has just installed a dining reservation application 505. Inresponse to the installation process the set identification engine maydetermine that the application 505 has expressly requested a particularfixed set of entities or, e.g., using a set prediction engine, is likelyto refer to the particular fixed set. Accordingly, the user may bepresented with confirmation window 510. Similar to confirmation window410, the confirmation window 510 may provide the opportunity for theuser to elect to download the particular slice or to reject thedownload. In some implementations, the application 505 may generatewindow 510. In some implementations (not shown) the applicationinstallation process or the set identification engine may automaticallydownload the particular fixed set of entities.

FIG. 6 illustrates a flow diagram of an example process 600 forgenerating and updating fixed sets of entities, in accordance withdisclosed implementations. Process 600 may be performed by apersonalized entity repository system, such as system 100 of FIG. 1 .Process 600 may be used to generate and maintain various slices of anentity repository, i.e., fixed sets of entities. The entity repositorymay be any large entity repository, for example entity repository 130 ofFIG. 1 . Process 600 may begin by generating location-based sets ofentities (605). A location-based set of entities may be any entities inan entity repository that are located in a particular cell. The cell maybe defined by boundaries, for example by lines connecting three or moregeographic coordinates. The cell may correspond to one or more geo cells(e.g., cells defined by a map application). The system may also generatetopic-based fixed sets of entities (610). Topic-based sets may be formedby clustering, by similarity with or connection to a seed entity, bymembership in a collection, etc. The system may also generate functionalsets of entities (615). The functional sets may be combinations oflocation-based sets and topic sets, may be based on use by a particularapplication, may be based on popularity (i.e., the most commonlysearched for entities and/or those appearing in the most crawleddocuments), may be based on a task or action, etc. Steps 605 to 615 maybe performed at an initial time and to update sets. For example, at aninitial time the system may perform steps 605 to 615 to generate initialsets of entities. Subsequently, the system may perform steps 605 to 615to update the initial sets, add new sets, etc.

The system may version the fixed sets of entities. For example, whenformat changes to the set would cause an application that uses the setsto fail, the system may keep older versions and generate a new versionafter the schema change. In this manner, the system may handle schemachanges in a manner that does not cause errors on client devices.Accordingly, the system may determine if the newly generated setsinclude a schema change (620). If so, the newly generated sets (e.g.,from steps 605 to 615), may be assigned a new version identifier (625).The version identifier will enable applications that use the sets tocorrectly download and use appropriate versions. No delta set is neededwhen the updates involve a schema change. If there is no schema change(620, No), the system may generate a delta for each set (630). Thedeltas are used when the client devices download just changes to fixedsets of entities rather than doing a delete of all fixed sets in thepersonalized entity model followed by storing the most recent sets ofhighest ranked fixed entity sets for that device. A delta may be a listof entities to remove and a list of entities to add. Thus, an update toan entity, e.g., name change, metadata or attribute change, etc., may bea delete of the entity followed by reading the entity. Deltas enable thesystem to transmit less data between the server where the sets aremaintained and the client devices where the sets are used. However, step630 is optional, and may not be performed for all sets or for even forany sets. For example, the client device, when updating the personalizedentity repository, may delete all currently stored slices and fetch thedesired slices (e.g., the predicted slices that fall within set usageparameters) and store those as the personalized entity model. Each fixedset of entities may have a unique identifier. In some implementations,the unique identifier may be a hash of the contents of the fixed set.Process 600 then ends. Process 600 may be repeated periodically toensure the fixed sets of entities are current.

FIG. 7 illustrates a flow diagram of an example process 700 for buildinga personal entity repository using pre-computed sets of entities, inaccordance with disclosed implementations. Process 700 may be performedby a personalized entity repository system, such as system 100 of FIG. 1. Process 700 may be performed on a client device to determine whichfixed set of entities or slices are included in a personalized entityrepository stored on the client device. The client device may performprocess 700 continuously, on a periodic basis, upon network connectivitychange, upon a triggering event, such as installation of a newapplication, an incoming phone call, addition of a new contact, etc., oruser request of a synchronization. Process 700 may be performed by a setidentification engine, such as set identification engine 162.

Process 700 may begin by identifying fixed sets of entities relevant toa user of a client device (705). The sets may be fixed in the sense thatthe sets are determined prior to performance of step 705 and not as partof step 705. In other words, the sets are determined independently ofany particular user or query to the entity repository. The setsidentified as part of step 705 are a much smaller subset than the totalnumber of fixed sets. The identification of the fixed sets may be based,for example, on context associated with the client device. For example aset prediction model may be provided a set of features generated from avariety of signals (i.e., context) that can include metadata andinformation collected from content generated on the client device, forexample via a content agent, search history, user profiles, etc. In someimplementations the information used as a signal may include informationfrom devices in proximity to the client device, e.g., from appliances orother computing devices in an Internet of Things home, from otherpersonal computing devices associated with the user or the user'shousehold, etc. Although such signals originate from another device, theproximity is context associated with the client device. In someimplementations, the information may include the location of thecomputing device, a timestamp, content in a search history for the user,content in a screen capture index, content associated with a userprofile, the applications installed on the client device, an activityperformed by the user (e.g., changing device settings, installing anapplication), etc. The signals may be provided to the set predictionmodel, such as set prediction model 122 or set prediction model 164,which may in turn provide one or more predicted fixed sets of entities.The fixed sets of entities identified in step 705 may also be based on alocation of the device, applications installed on the device, tasks oractivities commonly performed on the device, etc.

The system may then rank the identified fixed sets of entities byrelevancy (710). The relevancy may be based on the probability scoreprovided by the set prediction model. In some implementations, thesystem may adjust that probability based on other information, such aslocation of the device, time of day, other devices in proximity, etc.For example, if two traveling companions have linked their devices, thedevice of the first companion may determine, e.g., via device to devicesignaling, that the device of the second companion has a particularslice of the entity repository stored and the device of the firstcompanion may demote the relevancy of that slice, e.g., because one ofthe two devices already has that slice of the entity repository storedin the personalized entity repository. As another example, the user'sdevice may have a particular application that requires a particularslice of the entity repository and the system may boost the relevancy ofthat particular slice. As another example, the system may boost therelevancy of a slice that is specified by the user, e.g., via control225 of FIG. 2 . For location-based fixed sets of entities, the systemmay set a relevancy score for the set based on the distance of theclient device from the cell represented by the set. For example, when aclient device is located within the cell, the relevancy may be veryhigh. The relevancy of other cells may be based on the distance betweena center of the cell and the current location of the client device, sothat longer distances produce lower relevancy.

The system may determine selected sets for the personalized entityrepository based on the set usage parameters and the ranks (715). Insome implementations, the system may determine a relevancy thresholdbased on the set usage parameters and the ranks of the fixed entity setsidentified as relevant to the user. Any sets with a relevancy score thatmeets the threshold may be included in the sets selected for thepersonalized entity repository. FIG. 9 is another example of theselection of the sets based on the set usage parameters and rank. Thesystem may then determine whether the personalized entity repositoryneeds to be updated (720). For example, if the selected fixed sets ofentities are the same as the sets currently included in the personalizedentity repository, but the system has not recently checked for updates,then the system may update the repository (720, Yes). As anotherexample, if the system has been notified of an update to one of thefixed sets in the selected sets, the system may update the repository(720, Yes). If a fixed set of entities is in the selected sets but notcurrently in the personalized entity repository, the system may updatethe repository (720, Yes). If the system does not need to update therepository (720, No), process 700 ends.

If the system does update the repository (720, Yes), the system mayselect a highest ranked set of the selected sets (725). For ease ofexplanation this highest ranked set may be referred to as the first set.The system may determine if this first set is already in thepersonalized entity repository (730). If it is in the repository (730,Yes), the system may update the first set on the client device, if anupdate is needed (735). The system may determine if an update is neededby either pulling the first set from the server or by notification fromthe server that an update exists. The update may be in the form of adelta, e.g., a list of entities to delete and a list of entities to addto the fixed set. If the first set is not in the personalized entityrepository (730, No), the system may determine if the personal entityrepository has room for the first set (740). For example, the personalentity repository may have a limit on the number of sets in therepository or the memory used by the repository, or both. If theaddition of the first set would exceed the limits (740, No), the systemmay determine whether there is a fixed set currently in the personalizedentity repository that can be deleted (745). A set that is currently inthe personalized entity repository can be deleted if it is not in theselected fixed sets of entities, i.e., those identified in step 715. Aset can also be deleted if it has a lower rank that the first set. Ifthere is a set to delete (745, Yes), the system may delete the set fromthe personalized entity repository (750) and return to step 740. If nosets can be deleted (745, No), process 700 may end.

When there is room for the first set in the personalized entityrepository (740, yes), the system may add the first set to thepersonalized entity repository on the mobile device (755). The systemmay then determine whether there is a next highest ranked set in theselected sets (760). If there is a next highest set (760, Yes), thesystem may perform steps 725 to 760 for the next set. This next set thenbecomes the first set for the ease of explanation. Thus, for example,the system may determine if the next set, i.e., now the first set, isalready in the personalized entity repository (730), etc. When all setsin the selected sets have been processed via steps 730 to 760 (760, No),the system has updated the personalized entity repository and process700 ends.

FIG. 8 illustrates a flow diagram of an example process 800 foridentifying fixed sets of entities relevant to a user of a clientdevice, in accordance with disclosed implementations. Process 800 may beperformed by a personalized entity repository system, such as system 100of FIG. 1 . Process 800 may be performed on a client device as part ofstep 705 of FIG. 7 to determine which fixed set of entities or slicesare relevant to a user. Process 800 begins by determining a location ofthe client device (805). The location may be expressed as coordinates,e.g., Global Positioning System (GPS) coordinates, or as the identifierof a geo cell, or some other methods of expressing a location on theEarth. The system may assign the relevancy of a location-based fixed setof entities based on the distance from the client device to the cellrepresented by the location-based fixed set (810). The distance may bemeasured from the location of the client device to a center of the cell.The relevancy may also have an inverse relationship to the distance, sothat a shorter distance has a higher relevancy. When the client deviceis located in a cell for a particular fixed set of entities, theparticular fixed set of entities may receive a highest relevancy score.

The system may also determine topic-based fixed sets of entities thatare relevant to the user (815). The topic-based sets may be determinedwith a set prediction model, such as set prediction model 164 or setprediction model 122, that provides one or more identifiers for fixedsets of entities that are predicted based on features. The features maybe based on a number of information items, such as the location of thedevice, content in search records, a user profile, or screen capturecontent, data from devices in proximity to the client device, e.g.,exchanging data with the client device, a timestamp, applicationsinstalled or executing at the time, etc. The system may set therelevancy score of each topic-based fixed set of entities to asimilarity with an embedding for the user of the client device (820). Insome implementations, the similarity may be the probability associatedwith the fixed set provided by the set prediction model. The system mayalso determine functional fixed sets of entities (825). The functionalsets may be entities deemed most popular, e.g., most commonly searchedfor entities, most commonly accessed entities, entities found most oftenin crawled documents, etc. The functional sets may also include entitiesused for a particular task, action, or application. For example, afunctional set may include entities that enable the device to performOCR in a particular language, or that are used by an art appreciationapplication, etc. Functional sets may also include combinations oflocation-based and topic-based sets, for example, movie theaters inPortland Oregon or national monuments in Washington D.C. In someimplementations, the system may use entity popularity as a relevancyscore for a functional fixed set of entities. In some implementations, afixed set may be ranked high based on an application that the user hasinstalled. Process 800 then ends, having identified fixed sets ofentities relevant to the user of the computing device and assigning arelevance score to each set.

FIG. 9 illustrates a flow diagram of an example process 900 forselecting fixed sets of entities from fixed sets of entities relevant toa user of a client device based on set usage parameters, in accordancewith disclosed implementations. Process 900 may be performed by apersonalized entity repository system, such as system 100 of FIG. 1 .Process 900 may be performed on a client device as part of step 715 ofFIG. 7 to determine which fixed set of entities or slices that arerelevant to the user are selected for inclusion in a personal entityrepository based on set usage parameters and rank. Process 900 begins byinitializing set count and model size variables. The set count may beset to zero and the repository size may also be set to zero. The systemmay then select from the fixed sets identified as relevant the fixed setwith the highest rank (910). The system may determine whether the setusage parameters include a quantity parameter (915). The quantityparameter is the maximum sets 215 of FIG. 2 and represents a limit onthe number of fixed sets that can be in the personalized entityrepository. If there is a quantity parameter (915, Yes), the system mayincrease the set count (920) by one. The system may then determine ifthe set count is greater than the quantity parameter (925). If it is,process 900 may end because the fixed set would go beyond the limitsrepresented by the set usage parameters. However, for the highest rankedset, the set count is not greater than the quantity parameter (925, No),so the system continues with step 930.

The system determines if the set usage parameters include a spaceparameter (930). The space parameter may be expressed as a specifiedamount of memory (e.g., maximum storage 210 of FIG. 2 ) or a percentageof available memory (e.g., percentage of storage 220 of FIG. 2 ), or acombination of these. If the set usage parameters do include a spaceparameter (930, Yes), the system may add the size of the fixed set beingexamined to the repository size (953). The size may be known or may beestimated, for example, based on an average size of fixed sets ofentities. The system may determine whether the repository size isgreater than the space parameter (940). If it is (940, Yes), process 900ends because the fixed set would violate a limit established by the setusage parameters. Otherwise (940, No), the system ads the set to theselected sets (945). The selected sets are the highest ranked fixed setsidentified as relevant to the user that also meet limits established bythe set usage parameters. If there is another set in the identified sets(e.g., the fixed sets identified as relevant to the user), 950, Yes, thesystem may select the next highest ranked set (955) and continue withsteps 915 to 950 ad described above. If no other sets exist in theidentified sets (950, No), process 900 ends, having identified sets tobe included in the repository based on the set usage parameters.

FIG. 10 shows an example of a generic computer device 1000, which may beoperated as server 110, and/or client device 150 of FIG. 1 , which maybe used with the techniques described here. Computing device 1000 isintended to represent various example forms of computing devices, suchas laptops, desktops, workstations, personal digital assistants,cellular telephones, smartphones, tablets, servers, and other computingdevices, including wearable devices. The components shown here, theirconnections and relationships, and their functions, are meant to beexamples only, and are not meant to limit implementations of theinventions described and/or claimed in this document.

Computing device 1000 includes a processor 1002, memory 1004, a storagedevice 1006, and expansion ports 1010 connected via an interface 1008.In some implementations, computing device 1000 may include transceiver1046, communication interface 1044, and a GPS (Global PositioningSystem) receiver module 1048, among other components, connected viainterface 1008. Device 1000 may communicate wirelessly throughcommunication interface 1044, which may include digital signalprocessing circuitry where necessary. Each of the components 1002, 1004,1006, 1008, 1010, 1040, 1044, 1046, and 1048 may be mounted on a commonmotherboard or in other manners as appropriate.

The processor 1002 can process instructions for execution within thecomputing device 1000, including instructions stored in the memory 1004or on the storage device 1006 to display graphical information for a GUIon an external input/output device, such as display 1016. Display 1016may be a monitor or a flat touchscreen display. In some implementations,multiple processors and/or multiple buses may be used, as appropriate,along with multiple memories and types of memory. Also, multiplecomputing devices 1000 may be connected, with each device providingportions of the necessary operations (e.g., as a server bank, a group ofblade servers, or a multi-processor system).

The memory 1004 stores information within the computing device 1000. Inone implementation, the memory 1004 is a volatile memory unit or units.In another implementation, the memory 1004 is a non-volatile memory unitor units. The memory 1004 may also be another form of computer-readablemedium, such as a magnetic or optical disk. In some implementations, thememory 1004 may include expansion memory provided through an expansioninterface.

The storage device 1006 is capable of providing mass storage for thecomputing device 1000. In one implementation, the storage device 1006may be or include a computer-readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product can be tangibly embodied insuch a computer-readable medium. The computer program product may alsoinclude instructions that, when executed, perform one or more methods,such as those described above. The computer- or machine-readable mediumis a storage device such as the memory 1004, the storage device 1006, ormemory on processor 1002.

The interface 1008 may be a high speed controller that managesbandwidth-intensive operations for the computing device 1000 or a lowspeed controller that manages lower bandwidth-intensive operations, or acombination of such controllers. An external interface 1040 may beprovided so as to enable near area communication of device 1000 withother devices. In some implementations, controller 1008 may be coupledto storage device 1006 and expansion port 1014. The expansion port,which may include various communication ports (e.g., USB, Bluetooth,Ethernet, wireless Ethernet) may be coupled to one or more input/outputdevices, such as a keyboard, a pointing device, a scanner, or anetworking device such as a switch or router, e.g., through a networkadapter.

The computing device 1000 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 1030, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system. In addition, itmay be implemented in a computing device, such as a laptop computer1032, personal computer 1034, or tablet/smart phone 1036. An entiresystem may be made up of multiple computing devices 1000 communicatingwith each other. Other configurations are possible.

FIG. 11 shows an example of a generic computer device 1100, which may beserver 110 of FIG. 1 , which may be used with the techniques describedhere. Computing device 1100 is intended to represent various exampleforms of large-scale data processing devices, such as servers, bladeservers, datacenters, mainframes, and other large-scale computingdevices. Computing device 1100 may be a distributed system havingmultiple processors, possibly including network attached storage nodes,that are interconnected by one or more communication networks. Thecomponents shown here, their connections and relationships, and theirfunctions, are meant to be examples only, and are not meant to limitimplementations of the inventions described and/or claimed in thisdocument.

Distributed computing system 1100 may include any number of computingdevices 1180. Computing devices 1180 may include a server or rackservers, mainframes, etc. communicating over a local or wide-areanetwork, dedicated optical links, modems, bridges, routers, switches,wired or wireless networks, etc.

In some implementations, each computing device may include multipleracks. For example, computing device 1180 a includes multiple racks 1158a-1158 n. Each rack may include one or more processors, such asprocessors 1152 a-1152 n and 1162 a-1162 n. The processors may includedata processors, network attached storage devices, and other computercontrolled devices. In some implementations, one processor may operateas a master processor and control the scheduling and data distributiontasks. Processors may be interconnected through one or more rackswitches 1158, and one or more racks may be connected through switch1178. Switch 1178 may handle communications between multiple connectedcomputing devices 1100.

Each rack may include memory, such as memory 1154 and memory 1164, andstorage, such as 1156 and 1166. Storage 1156 and 1166 may provide massstorage and may include volatile or non-volatile storage, such asnetwork-attached disks, floppy disks, hard disks, optical disks, tapes,flash memory or other similar solid state memory devices, or an array ofdevices, including devices in a storage area network or otherconfigurations. Storage 1156 or 1166 may be shared between multipleprocessors, multiple racks, or multiple computing devices and mayinclude a computer-readable medium storing instructions executable byone or more of the processors. Memory 1154 and 1164 may include, e.g.,volatile memory unit or units, a non-volatile memory unit or units,and/or other forms of computer-readable media, such as a magnetic oroptical disks, flash memory, cache, Random Access Memory (RAM), ReadOnly Memory (ROM), and combinations thereof. Memory, such as memory 1154may also be shared between processors 1152 a-1152 n. Data structures,such as an index, may be stored, for example, across storage 1156 andmemory 1154. Computing device 1100 may include other components notshown, such as controllers, buses, input/output devices, communicationsmodules, etc.

An entire system, such as server 110, may be made up of multiplecomputing devices 1100 communicating with each other. For example,device 1180 a may communicate with devices 1180 b, 1180 c, and 1180 d,and these may collectively be known as server 110. As another example,system 100 of FIG. 1 may include one or more computing devices 1100.Some of the computing devices may be located geographically close toeach other, and others may be located geographically distant. The layoutof system 1100 is an example only and the system may take on otherlayouts or configurations.

According to certain aspects of the disclosure, a mobile devicecomprises a display device, a personalized entity repository stored inmemory, the personalized entity repository including a plurality offixed sets of entities from an entity repository stored at a server,wherein each fixed set has a respective identifier and includesinformation about the entities in the set, at least one processor, andmemory storing instructions that, when executed by the at least oneprocessor, cause the mobile device to perform operations. The operationinclude identifying fixed sets of the entity repository that arerelevant to a user of the mobile device based on context associated withthe mobile device, ranking the fixed sets by relevancy, determiningselected sets from the identified fixed sets using the rank and setusage parameters applicable to the user, and updating the personalizedentity repository using the selected sets.

This and other aspects can include one or more of the followingfeatures. For example, updating the personalized entity repository mayoccur responsive to determining that a first fixed set of the identifiedfixed sets does not exist in the personalized entity repository. Asanother example, updating the personalized entity repository can includeremoving a set in the personalized entity repository that is not aselected set. As another example, the set usage parameters can include aquantity of fixed sets and/or an amount of memory allocated to thepersonalized entity repository, the amount being set by the user. Asanother example, the plurality of fixed sets stored in the personalizedentity repository can include location sets, wherein entities in alocation set are located in a same cell. As another example, theplurality of fixed sets stored in the personalized entity repositoryincludes topic sets, wherein entities in a topic set are entitiesclustered together based on entity characteristics. In someimplementations, the rank of a first topic set is a rank assigned by aprediction model based on items recognized in content generated fordisplay on the display device. As another example, updating thepersonalized entity repository can include adding a fixed set that doesnot currently exist in the personalized entity repository and using adelta of a fixed set that exists in the personalized entity repository.

As another example, the operations may also include identifying anentity in content generated for display on the display device using thepersonalized entity repository. As another example, each of theplurality of sets has a version identifier, the version identifierchanging when an update to a particular set breaks a schema of thepersonalized entity repository. As another example, the operations mayalso include initiate display of information that identifies a firstfixed set from the selected sets and a control configured to enable theuser to accept instillation of the first fixed set and download thefirst fixed set and add it to the personalized entity repository whenthe user accepts installation. As another example, identifying fixedsets of the entity repository that are relevant to the user based on thecontext can include determining a location of the mobile device andusing the location to determine location-based fixed sets relevant tothe user, using a set prediction model to identify topic-based setsrelevant to the user, and determining sets identified by at least oneapplication installed on the mobile device. As another example,identifying fixed sets of the entity repository that are relevant to theuser based on the context can include determining content previouslyviewed on the mobile device and providing the content previously viewedto a set prediction model configured to predict at least one topic-basedset from the content previously viewed and/or determining recentsearches conducted on the mobile device and providing the content fromthe recent searches to a set prediction model configured to predict atleast one topic-based set from the content. As another example,identifying fixed sets of the entity repository that are relevant to theuser based on the context can include determining an activity performedby the user of the mobile device and predicting a fixed set of entitiesbased on the activity.

According to certain aspects of the disclosure, a method includesreceive a screen captured image configured to display content on adisplay of a mobile device, determining text in the image by performingtext recognition on the image, providing the text to a set predictionmodel, wherein the set prediction model is trained to predict one ormore fixed sets of entities, and storing at least one fixed set ofentities of the predicted fixed sets of entities in a personalizedentity repository in memory on the mobile device.

These and other aspects can include one or more of the followingfeatures. For example, the method may also include identifying entitiesin screen capture images using the personalized entity repository. Themethod may also include determining a location associated with contentrecognized in a screen capture image, provide the location to the setprediction model, the set prediction model predicting at least onelocation-based fixed set of entities based on the location, and storingthe location-based fixed set of entities in the personalized entityrepository. As another example, the prediction model may predict aplurality of predicted fixed sets of entities and the method alsoincludes ranking each of the predicted fixed sets of entities, using setusage parameters and the rankings to determine selected fixed sets ofentities from the predicted fixed sets of entities and storing theselected fixed sets of entities in the personalized entity repository.

According to certain aspects of the disclosure, a method may includegenerating a plurality of fixed sets of entities from an entityrepository. The fixed sets include location-based sets, eachlocation-based set including entities from the entity repository thathave a location inside a cell, the cell being associated with the set,and topic-based sets, at least some of the topic-based sets includingentities from the entity repository that are related to each other viaan embedding similarity. The method may also include providing a subsetof the fixed sets of entities to a client device, the client devicerequesting the subset based on a location of the client device and onrecognized items identified in content generated for display on theclient device.

These and other aspects can include one or more of the followingfeatures. For example, the method may also include identifying at leastone entity added to the entity repository, determining at least onefixed set that the added entity belongs to, and generating a delta forthe at least one fixed set that includes the added entity. As anotherexample, the method may also include identifying at least one entityadded to the entity repository, determining at least one fixed set thatthe added entity belongs to, determining that a schema of the fixed setshas changed, and generating a new version of the at least one fixed set,the new version including the added entity and having a new versionidentifier. As another example, at least some of the topic-based setsmay include entities similar to a seed entity and/or the fixed sets mayalso include functional sets, at least one of the functional setsincluding entities deemed most popular.

Various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium”“computer-readable medium” refers to any non-transitory computer programproduct, apparatus and/or device (e.g., magnetic discs, optical disks,memory (including Read Access Memory), Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

A number of implementations have been described. Nevertheless, variousmodifications may be made without departing from the spirit and scope ofthe invention. In addition, the logic flows depicted in the figures donot require the particular order shown, or sequential order, to achievedesirable results. In addition, other steps may be provided, or stepsmay be eliminated, from the described flows, and other components may beadded to, or removed from, the described systems. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A method implemented by one or more processors,the method comprising: determining that each of multiple userinteractions via a user device relate to a particular topic;determining, based on the multiple user interactions that each relate tothe particular topic, a confidence score for a topic-based set ofentities for the particular topic; in response to the confidence scorefor the topic-based set of entities satisfying a threshold: causing aninteractive prompt, related to the topic-based set of entities, to berendered at the user device; in response to an acceptance userinteraction with the interactive prompt at the user device: causing theuser device to download the topic-based set of entities; and in responseto a rejection user interaction with the interactive prompt at the userdevice: refraining from causing the user device to download thetopic-based set of entities.
 2. The method of claim 1, furthercomprising: subsequent to causing the user device to download thetopic-based set of entities: identifying an additional user interactionvia the user device related to an application that is currently runningon the user device; determining that the additional user interaction isassociated with at least one entity from the topic-based set ofentities; selecting, based on determining that the additional userinteraction is associated with at least one entity from the topic-basedset of entities, one or more actions to suggest to a user of the userdevice; and causing the user device to display one or more suggestedactions based on the one or more selected actions.
 3. The method ofclaim 2, wherein the one or more suggested actions that are displayedare each associated with a corresponding different application that isdifferent from the application related to the additional userinteraction.
 4. The method of claim 2, wherein each of the one or moresuggested actions is associated with at least one additionalcorresponding entity from the topic-based set of entities.
 5. The methodof claim 1, wherein determining that each of the multiple userinteractions via the user device relate to the particular topiccomprises: generating multiple embeddings for the multiple userinteractions, each of the multiple embeddings being generated based onone or more of: an application currently in use by the user during arespective one of the multiple user interactions, screen metadataindicating content displayed on a screen of the user device during therespective one of the multiple user interactions, and/or contentassociated with recent search queries submitted by the user as indicatedby a search log; comparing the multiple embeddings to a plurality oftopic embeddings; and determining, based on the comparing, that each ofthe multiple user interactions via the user device relate to theparticular topic.
 6. The method of claim 1, wherein determining thateach of the multiple user interactions via the user device relate to theparticular topic comprises: generating multiple embeddings for themultiple user interactions, each of the multiple embeddings beinggenerated based on content associated with recent search queriessubmitted by the user as indicated by a search log; comparing themultiple embeddings to a plurality of topic embeddings; and determining,based on the comparing, that each of the multiple user interactions viathe user device relate to the particular topic.
 7. The method of claim1, wherein determining that each of the multiple user interactions viathe user device relate to the particular topic comprises: generatingmultiple embeddings for the multiple user interactions, each of themultiple embeddings being generated based on screen metadata indicatingcontent displayed on a screen of the user device during the respectiveone of the multiple user interactions; comparing the multiple embeddingsto a plurality of topic embeddings; and determining, based on thecomparing, that each of the multiple user interactions via the userdevice relate to the particular topic.
 8. The method of claim 1, whereindetermining the confidence score for the topic-based set of entities forthe particular topic comprises: determining the confidence score using atrained set prediction model and features related to one or more of themultiple user interactions.
 9. A system comprising: one or morecomputing devices and more or more storage devices storing instructionsthat are operable, when executed by the one or more computing devices,to cause the one or more computing devices to perform operationscomprising: determining that each of multiple user interactions via auser device relate to a particular topic; determining, based on themultiple user interactions that each relate to the particular topic, aconfidence score for a topic-based set of entities for the particulartopic; in response to the confidence score for the topic-based set ofentities satisfying a threshold: causing an interactive prompt, relatedto the topic-based set of entities, to be rendered at the user device;in response to an acceptance user interaction with the interactiveprompt at the user device: causing the user device to download thetopic-based set of entities; and in response to a rejection userinteraction with the interactive prompt at the user device: refrainingfrom causing the user device to download the topic-based set ofentities.
 10. The system of claim 9, the operations further comprising:subsequent to causing the user device to download the topic-based set ofentities: identifying an additional user interaction via the user devicerelated to an application that is currently running on the user device;determining that the additional user interaction is associated with atleast one entity from the topic-based set of entities; selecting, basedon determining that the additional user interaction is associated withat least one entity from the topic-based set of entities, one or moreactions to suggest to a user of the user device; and causing the userdevice to display one or more suggested actions based on the one or moreselected actions.
 11. The system of claim 10, wherein the one or moresuggested actions displayed are each associated with a correspondingdifferent application that is different from the application related tothe additional user interaction.
 12. The system of claim 10, whereineach of the one or more suggested actions is associated with at leastone additional corresponding entity from the topic-based set ofentities.
 13. The system of claim 9, wherein determining that each ofmultiple user interactions via a user device relate to a particulartopic comprises: generating multiple embeddings for the multiple userinteractions, each of the multiple embeddings being generated based onone or more of: an application currently in use by the user during arespective one of the multiple user interactions, screen metadataindicating content displayed on a screen of the user device during therespective one of the multiple user interactions, and/or contentassociated with recent search queries submitted by the user as indicatedby a search log; comparing the multiple embeddings to a plurality oftopic embeddings; and determining, based on the comparing, that each ofthe multiple user interactions via the user device relate to theparticular topic.
 14. The system of claim 9, wherein determining thateach of multiple user interactions via a user device relate to aparticular topic comprises: generating multiple embeddings for themultiple user interactions, each of the multiple embeddings beinggenerated based on content associated with recent search queriessubmitted by the user as indicated by a search log; comparing themultiple embeddings to a plurality of topic embeddings; and determining,based on the comparing, that each of the multiple user interactions viathe user device relate to the particular topic.
 15. The system of claim9, wherein determining that each of multiple user interactions via auser device relate to a particular topic comprises: generating multipleembeddings for the multiple user interactions, each of the multipleembeddings being generated based on an application currently in use bythe user during a respective one of the multiple user interactions;comparing the multiple embeddings to a plurality of topic embeddings;and determining, based on the comparing, that each of the multiple userinteractions via the user device relate to the particular topic.
 16. Thesystem of claim 9, wherein determining the confidence score for thetopic-based set of entities for the particular topic comprises:determining the confidence score using a trained set prediction modeland features related to one or more of the multiple user interactions.